<html>
<head><meta charset="utf-8"><title>meeting 2021-03-29 · t-infra · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/index.html">t-infra</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html">meeting 2021-03-29</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="232299293"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232299293" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232299293">(Mar 29 2021 at 16:53)</a>:</h4>
<p>I personally feel like the meeting is in 7 minutes, but if it's actually in 1 hour and 7 minutes I won't be able to make it</p>



<a name="232299815"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232299815" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232299815">(Mar 29 2021 at 16:56)</a>:</h4>
<p><span class="user-group-mention" data-user-group-id="2943">@T-infra</span> T-4 minutes</p>



<a name="232300406"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232300406" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232300406">(Mar 29 2021 at 17:00)</a>:</h4>
<p><span class="user-group-mention" data-user-group-id="2943">@T-infra</span> meeting time!</p>



<a name="232300437"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232300437" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Jake Goulding <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232300437">(Mar 29 2021 at 17:00)</a>:</h4>
<p>o/</p>



<a name="232300443"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232300443" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232300443">(Mar 29 2021 at 17:00)</a>:</h4>
<p>o/</p>



<a name="232300450"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232300450" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> kennytm <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232300450">(Mar 29 2021 at 17:00)</a>:</h4>
<p>O/</p>



<a name="232300655"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232300655" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232300655">(Mar 29 2021 at 17:02)</a>:</h4>
<p>o/</p>



<a name="232300762"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232300762" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232300762">(Mar 29 2021 at 17:02)</a>:</h4>
<p>ok let's start!</p>



<a name="232300791"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232300791" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232300791">(Mar 29 2021 at 17:03)</a>:</h4>
<p>there is one nominated issue:  add dynamically-linked musl targets <a href="https://github.com/rust-lang/rust/issues/82556">#82556</a></p>



<a name="232300886"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232300886" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232300886">(Mar 29 2021 at 17:03)</a>:</h4>
<p>this seems like to be nominated for T-compiler though</p>



<a name="232300902"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232300902" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232300902">(Mar 29 2021 at 17:03)</a>:</h4>
<p>anyone opposed to skipping?</p>



<a name="232301017"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301017" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301017">(Mar 29 2021 at 17:04)</a>:</h4>
<p>ok let's skip</p>



<a name="232301029"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301029" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301029">(Mar 29 2021 at 17:04)</a>:</h4>
<p>there are no other issues for us to look at</p>



<a name="232301071"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301071" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301071">(Mar 29 2021 at 17:04)</a>:</h4>
<p>the only item on our agenda is the following by me</p>



<a name="232301079"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301079" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301079">(Mar 29 2021 at 17:04)</a>:</h4>
<blockquote>
<p><a href="http://crates.io">crates.io</a> access to prometheus</p>
</blockquote>



<a name="232301264"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301264" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301264">(Mar 29 2021 at 17:05)</a>:</h4>
<p>the <a href="http://crates.io">crates.io</a> team wants to start collecting more metrics to aid with running <a href="http://crates.io">crates.io</a>, and it'd be nice to use our central prometheus instance for that</p>



<a name="232301362"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301362" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301362">(Mar 29 2021 at 17:06)</a>:</h4>
<p>this is the monitoring aws ec2 machine, correct? that also serves <a href="http://grafana.rust-lang.org">grafana.rust-lang.org</a>?</p>



<a name="232301383"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301383" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301383">(Mar 29 2021 at 17:06)</a>:</h4>
<p>yes</p>



<a name="232301401"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301401" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301401">(Mar 29 2021 at 17:06)</a>:</h4>
<p>this would of course mean allowing the <a href="http://crates.io">crates.io</a> team to access our grafana instance, but there is also some extra infra changes that could be required</p>



<a name="232301485"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301485" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301485">(Mar 29 2021 at 17:07)</a>:</h4>
<p>namely, standard prometheus doesn't work on heroku, because prometheus needs to access the single instances directly for the metrics we want to collect, but heroku doesn't expose the singlel instances at all</p>



<a name="232301578"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301578" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301578">(Mar 29 2021 at 17:07)</a>:</h4>
<p>one of the ways the <a href="http://crates.io">crates.io</a> team is investigating is to run a prometheus instance on <em>every</em> <a href="http://crates.io">crates.io</a> dyno, configured to forward all metrics to our main prometheus instance</p>



<a name="232301639"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301639" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301639">(Mar 29 2021 at 17:08)</a>:</h4>
<p>that'd require to expose the prometheus remote write api to the internet</p>



<a name="232301673"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301673" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301673">(Mar 29 2021 at 17:08)</a>:</h4>
<p>my plan to experiment was to expose it token-protected</p>



<a name="232301684"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301684" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301684">(Mar 29 2021 at 17:08)</a>:</h4>
<p>in something like <a href="http://prometheus.infra.rust-lang.org">prometheus.infra.rust-lang.org</a></p>



<a name="232301699"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301699" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301699">(Mar 29 2021 at 17:08)</a>:</h4>
<p>any thoughts on this?</p>



<a name="232301771"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301771" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301771">(Mar 29 2021 at 17:09)</a>:</h4>
<p>I have a couple of questions</p>



<a name="232301825"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301825" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301825">(Mar 29 2021 at 17:09)</a>:</h4>
<ol>
<li>what's the upside? i.e. you mentioned that you want more metrics - is there something in particular that is motivating it?</li>
<li>has anyone else ever done this?</li>
</ol>



<a name="232301957"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232301957" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232301957">(Mar 29 2021 at 17:10)</a>:</h4>
<blockquote>
<p>what's the upside? i.e. you mentioned that you want more metrics - is there something in particular that is motivating it?</p>
</blockquote>
<p>a metric that would be incredibly useful is the number of available connections in the database pool</p>



<a name="232302116"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232302116" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232302116">(Mar 29 2021 at 17:11)</a>:</h4>
<p>this would be for alerting reasons?</p>



<a name="232302243"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232302243" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232302243">(Mar 29 2021 at 17:12)</a>:</h4>
<p>potentially, but it would also be useful for looking at the health of the service while investigating other issues</p>



<a name="232302312"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232302312" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232302312">(Mar 29 2021 at 17:13)</a>:</h4>
<p>what the <a href="http://crates.io">crates.io</a> team is doing right now is selectively adding datapoints to log lines, but that is a bad experience when debugging live issues: you have to eyeball it and you might lose interesting datapoints as the logs move so fast</p>



<a name="232302328"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232302328" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232302328">(Mar 29 2021 at 17:13)</a>:</h4>
<p>this seems like an incredibly common desire -- exporting prometheus metrics from heroku dynos -- I imagine there are blog posts or other articles out there about it; to what extent is there a doc/scratch tabs/whatever with a collection of research done on this?</p>



<a name="232302398"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232302398" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232302398">(Mar 29 2021 at 17:13)</a>:</h4>
<p>at the very least, I'd like to better understand what is the commonly done thing before we decide to run prometheus on every instance (for example)</p>



<a name="232302420"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232302420" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232302420">(Mar 29 2021 at 17:13)</a>:</h4>
<p>or see some documentation of alternatives</p>



<a name="232302483"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232302483" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232302483">(Mar 29 2021 at 17:14)</a>:</h4>
<p>honestly there isn't much information on this that solves the problem in a great way</p>



<a name="232302490"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232302490" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Gaynor <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232302490">(Mar 29 2021 at 17:14)</a>:</h4>
<p>Rather than running a prometheus instance in the dynos, can't the "push gateway" be used for this?</p>



<a name="232302562"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232302562" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232302562">(Mar 29 2021 at 17:15)</a>:</h4>
<p>pushgateway is one of the usually suggested things, but one of its issues is that it doesn't support TTLs for the metrics it exposes</p>



<a name="232302625"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232302625" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232302625">(Mar 29 2021 at 17:15)</a>:</h4>
<p>so unless someone manually calls the API to clear up old metrics they will continue to report the same things</p>



<a name="232302774"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232302774" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232302774">(Mar 29 2021 at 17:16)</a>:</h4>
<p>(reading)</p>



<a name="232302779"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232302779" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232302779">(Mar 29 2021 at 17:16)</a>:</h4>
<p>I guess I'd love to see this information reflected in a less ephemeral shape, it seems important to document why we didn't go with the pushgateway, presuming we go with the solution you're suggesting; I imagine there are pros and cons</p>



<a name="232302868"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232302868" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232302868">(Mar 29 2021 at 17:16)</a>:</h4>
<p>the other alternatives that come up online are:</p>
<ul>
<li>recommendations not to do it</li>
<li>output metrics in the logs and have another service scrape the logs and ingest them into prometheus</li>
<li>have prometheus scrape the load balancer and <em>hope</em> it will eventually reach all nodes</li>
</ul>



<a name="232302934"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232302934" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232302934">(Mar 29 2021 at 17:17)</a>:</h4>
<p>ok, well, I guess - can you give some hint as to urgency? does this need a solution in the next week? next few months?</p>



<a name="232303079"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232303079" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232303079">(Mar 29 2021 at 17:18)</a>:</h4>
<p>(I would love to see those options in a doc -- e.g., recommendations not to do it, why? what problems do we expect)</p>



<a name="232303094"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232303094" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232303094">(Mar 29 2021 at 17:18)</a>:</h4>
<p>it's not something that we <em>have</em> to get ready asap, but the earlier it's available the earlier <a href="http://crates.io">crates.io</a> ops will improve</p>



<a name="232303153"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232303153" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232303153">(Mar 29 2021 at 17:18)</a>:</h4>
<p><span class="user-mention silent" data-user-id="121055">Pietro Albini</span> <a href="#narrow/stream/242791-t-infra/topic/meeting.202021-03-29/near/232302868">said</a>:</p>
<blockquote>
<ul>
<li>output metrics in the logs and have another service scrape the logs and ingest them into prometheus</li>
</ul>
</blockquote>
<p>this one gives me positive vibes because it sounds noninvasive on current infra</p>



<a name="232303295"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232303295" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232303295">(Mar 29 2021 at 17:19)</a>:</h4>
<p>well, it may also have downsides in terms of authenticating to those logs and how costly that scraping is (I just don't know) -- I guess this is why I'd personally feel much happier with some investigation done on each option before we choose one (or at least better understanding the costs of the proposed solution, in terms of engineering time)</p>



<a name="232303411"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232303411" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232303411">(Mar 29 2021 at 17:20)</a>:</h4>
<p>I was going to ask, I wasn't clear where this proposal was up to - it sounded like it was still at a kinda investigation stage</p>



<a name="232303534"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232303534" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232303534">(Mar 29 2021 at 17:21)</a>:</h4>
<p>definitely also curious :)</p>



<a name="232303539"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232303539" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232303539">(Mar 29 2021 at 17:21)</a>:</h4>
<p>yeah, this is still in investigation stage</p>



<a name="232303591"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232303591" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232303591">(Mar 29 2021 at 17:21)</a>:</h4>
<p>fair enough</p>



<a name="232303694"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232303694" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232303694">(Mar 29 2021 at 17:22)</a>:</h4>
<p>I'd like to do a proof of concept of the prometheus-on-dynos idea to evaluate whether it's viable (mostly that it doesn't use many resources on dynos)</p>



<a name="232303731"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232303731" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232303731">(Mar 29 2021 at 17:22)</a>:</h4>
<p>and that requires setting up the remote write api on <a href="http://monitoring.infra.rust-lang.org">monitoring.infra.rust-lang.org</a> to test whether it works</p>



<a name="232303809"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232303809" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232303809">(Mar 29 2021 at 17:23)</a>:</h4>
<p>is the remote write api a hardened thing, i.e. do other people expose it to the public internet?</p>



<a name="232303844"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232303844" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232303844">(Mar 29 2021 at 17:23)</a>:</h4>
<p>hm, I guess I don't have a good understanding of what that means (either step) - is there some documentation you can point to?</p>



<a name="232304000"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232304000" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232304000">(Mar 29 2021 at 17:24)</a>:</h4>
<p>all prometheus apis are completly unauthenticated, what I would do is to put nginx with authentication in front of it</p>



<a name="232304104"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232304104" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232304104">(Mar 29 2021 at 17:24)</a>:</h4>
<p>any other solution would require the same level of protection (nginx with http auth), as neither prometheus nor pushway implement any sort of auth internally</p>



<a name="232304185"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232304185" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232304185">(Mar 29 2021 at 17:25)</a>:</h4>
<p>(that's an explicit design goal of them iirc)</p>



<a name="232304208"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232304208" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232304208">(Mar 29 2021 at 17:25)</a>:</h4>
<p>yeah, I mean, understandable to some extent</p>



<a name="232304323"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232304323" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232304323">(Mar 29 2021 at 17:26)</a>:</h4>
<p>(sadface was mainly around the idea of http auth, not the end of the world - e.g. no ratelimiting by default)</p>



<a name="232304374"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232304374" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232304374">(Mar 29 2021 at 17:26)</a>:</h4>
<p>I am feeling like even the experimentation has a lot of moving parts and I would personally like to see a general layout/plan before moving ahead with it</p>



<a name="232304465"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232304465" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232304465">(Mar 29 2021 at 17:27)</a>:</h4>
<p>my summarised thoughts are: nervousness at the idea of treading new ground, I like scraping logs idea</p>



<a name="232304586"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232304586" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232304586">(Mar 29 2021 at 17:28)</a>:</h4>
<p>I have not done a full investigation yet, but my understanding is that scraping logs would have way more moving parts and would cost us more</p>



<a name="232304715"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232304715" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232304715">(Mar 29 2021 at 17:29)</a>:</h4>
<p>those are off the critical path of <a href="http://crates.io">crates.io</a> service I assume? i.e. if it breaks, monitoring breaks</p>



<a name="232304751"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232304751" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232304751">(Mar 29 2021 at 17:29)</a>:</h4>
<p>(which, notably may lead to paging, so it's not 'entirely noncritical', imo)</p>



<a name="232304814"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232304814" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232304814">(Mar 29 2021 at 17:29)</a>:</h4>
<p>yeah eventually I'd love to hook prometheus to pagerduty, so metrics breaking are not great</p>



<a name="232305026"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232305026" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232305026">(Mar 29 2021 at 17:31)</a>:</h4>
<p>definitely - let me restate to "does <a href="http://crates.io">crates.io</a> work isn't dependent on log scraping"</p>



<a name="232305074"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232305074" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232305074">(Mar 29 2021 at 17:31)</a>:</h4>
<p>yes</p>



<a name="232305398"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232305398" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232305398">(Mar 29 2021 at 17:33)</a>:</h4>
<p>anyway, I'll prepare a summary of the solutions in the future</p>



<a name="232305418"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232305418" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Aidan Hobson Sayers <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232305418">(Mar 29 2021 at 17:33)</a>:</h4>
<p>I don't have any follow up thoughts - I'm keen to continue hearing where it's up to</p>



<a name="232305487"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232305487" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232305487">(Mar 29 2021 at 17:33)</a>:</h4>
<p><span class="user-mention" data-user-id="261224">@triagebot</span> end-meeting</p>



<a name="232305490"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232305490" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> triagebot <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232305490">(Mar 29 2021 at 17:33)</a>:</h4>
<p>Does anyone have something to bring up?<br>
React with <span aria-label="working on it" class="emoji emoji-1f6e0" role="img" title="working on it">:working_on_it:</span> if you have something to say.<br>
React with <span aria-label="all good" class="emoji emoji-2705" role="img" title="all good">:all_good:</span> if you're ready to end the meeting.</p>



<a name="232305650"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232305650" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232305650">(Mar 29 2021 at 17:34)</a>:</h4>
<p>to be clear, I don't think there's any reason to block experimentation with your proposal on the general summary, but I would like to see a sketch (maybe just 5 bullet points) of the affected services and shape before you go ahead with the experimentation -- it may be that the proposal for experimentation is not approved after that, though, depending on what they are :)</p>



<a name="232305813"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232305813" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232305813">(Mar 29 2021 at 17:35)</a>:</h4>
<p>thanks again for attending then!</p>



<a name="232305815"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232305815" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Pietro Albini <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232305815">(Mar 29 2021 at 17:35)</a>:</h4>
<p>\o</p>



<a name="232305839"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232305839" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> kennytm <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232305839">(Mar 29 2021 at 17:36)</a>:</h4>
<p>\o</p>



<a name="232305952"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/242791-t-infra/topic/meeting%202021-03-29/near/232305952" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> simulacrum <a href="https://rust-lang.github.io/zulip_archive/stream/242791-t-infra/topic/meeting.202021-03-29.html#232305952">(Mar 29 2021 at 17:36)</a>:</h4>
<p>\o</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>